Recently, methods of performing object identification and the like making use of machine learning are being studied. As one of the machine learning, a deep learning, which uses a neural network with several hidden layers between an input layer and an output layer, shows high recognizing performance.
And, the neural network using the deep learning generally learns through backpropagation using losses.
In order to perform learning of such a deep learning network, training data in which tags, i.e., labels, are added to individual data points by labelers are needed. Preparing this training data (i.e. classifying the data correctly) can be very labour-intensive, expensive and inconvenient, especially if a large amount of training data is to be used and if the quality of the data pre-preparation is not consistently high. Conventional interactive labeling can be computationally expensive and fail to deliver good results.
Therefore, recently, auto-labeling which adds tags to training images using a deep learning-based auto labeling device is performed, and then inspectors examine auto-labeled training images to correct the tags or the labels.
Also, a reliability of labeling by the auto-labeling device and labelers is evaluated by adding validation images with their own true labels.
However, in a conventional method of adding the validation images to unlabeled images, if shooting environment like its neighborhood, weather, or day/night, of the unlabeled images and that of the validation images are different, the inspectors can easily distinguish between the unlabeled images and the validation images, and as a result, the inspection is prone to be performed mainly on the validation images, and labeling is not evaluated properly.
Also, if the inspection is done on the true labels only, as in the conventional method, lazy inspectors may be evaluated as higher than inspectors who work hard.
In addition, if labels of misdetection and non-detection are added to the true labels for validation, they may be different from the auto-labels by the auto-labeling device, in which case the inspectors can easily find out whether they are labels for validation, thus proper estimation is hampered.